TITLE OF THE INVENTION 

A Method for Lighting- and View- angle-invariant Face Description 
with first- and second-order Eigenfeatures 



BACKGROUND OF THE INVENTION 

This invention relates to a method for lighting- and view-angle-invariant face 
description with first- and second-order eigenfeatures, and can be used in face description 
and recognition for content-based image retrieval, huma face identification and verification for 
bank, security system, and video phone, surveillance and tracking, digital library, and internet 
multimedia database. 

Human face perception is an active area in the computer vision community. Face 
recognition will play an important role in multimedia database search and many other 
applications. In recent years considerable progress has been made on the problems of face 
detection and recognition. Different specific techniques were proposed. Among these, neural 
nets, elastic template matching, Karhunen-Loeve expansion, algebraic moments and isodensity 
lines are typical methods. 

Among these methods, Principal component analysis (PCA) or Karhunen-Loeve 
expansion is an important branch. Eigenface method is derived from PCA and it is convenient 
to be computed and has consistent accuracy in identification. Prior work showed that the PCA 
approach dissociates spontaneously between different types of information. The eigenvectors 
with large eigenvalues capture information that is common to subsets of faces and eigenvectors 
with small eigenvalues capture information specific to individual face. The studies show that 
only the information contained by the eigenvectors with large eigenvalues can be generalized to 
new faces that are not trained. 

The advantage of eigenface method is that eigenvectors with large eigenvalues convey 
information relative to the basic shape and structure of the faces. That means features extracted 
from eigenvectors with large eigenvalues can be used to describe major characteristics of human 
faces. However, this is also the weakness of PCA. If we only consider the features extracted 
from eigenvectors with large eigenvalues, we cannot get the details of the faces which 
corresponds to individual faces. If these details of individual face can be described with the 
common features of human faces, the description of human faces can be more accurate. 



Furthermore, it is difficult to get suitable eigenvectors to represent human faces if the 
face images are taken under different lighting conditions and different view-angles. According 
to our experience, the regular eigenface method is not excellent in recognizing faces under 
5 different lighting condition. However, many face images of the same person are taken under 
different lighting and view-angle conditions. The lighting-invariant and view-angle-invariant 
face description is still an important field to be explored. 

SUMMARY OF THE INVENTION 

U Eigenface method is effective to extract common face characteristics such as shape 

pj; and structure. In order to get details of faces that are lost when eigenvectors with small 

O eigenvalues are truncated, the reconstructed faces with the features from eigenvectors with large 

eigenvalues should be obtained. With the reconstructed face images, the residue images 
yj 15 between original images and reconstructed images can be obtained. These residue faces can be 
E looked as high-passed face images which still contain rich detailed information for individual 

kj faces. In order to describe these residue faces, eigenface method can be used on these residue 

p= faces again. The obtained eigenvectors with large eigenvalues will reveal the common 

2: characteristics of residue faces. With this method, the second-order eigenvectors with large 

|=4 20 eigenvalues can be obtained to extract corresponding features. According to our experiments, 
first-order and second-order eigenfeatures are both effective for the view-angle-invariant face 
description, while the second-order eigenfeatures are also effective for the lighting-invariant 
face description. Therefore, the combination of first-order and second-order eigenfeatures can 
be used for view-angle-invariant face description and the second-order eigenfeatures can be 
25 used for lighting-invariant face description. Another way to describe face is to use first-order 
eigenfeatures exclusively. For lighting-invariant face description, only the eigenfeatures 
extracted from the eigenfaces with the top k-th to N-th largest eigenvalues (where 0<k<N) to 
avoid the lighting variance coming from the top k eigenfaces with largest eigenvalues. The 
view-angle-invariant face features can be the eigenfeatures extracted from the eigenfaces 
30 corresponding to the top N largest eigenvalues. 

Besides, when we consider face description, an important characteristic is 
symmetry of faces. The faces of most of the people are symmetric. Thus we can 
consider to use the left-right mirrored face image to represent the same person. 
35 Consequently the eigenfeatures of the original image and the mirrored image can 
both represent the same person. Therefore, we can use the linear combination of the 



2 



eigenfeatures of the two images to represent the original image. In order to 
discriminate between the original image and the mirrored image, different weights 
can be used to combine the eigenfeatures. 



5 The present invention provides two methods for lighting-invariant and view-angle- 

invariant human face description which can be used for image retrieval (query by face example)., 
person identification and verification, surveillance and tracking, and other face recognition 
applications. In order to describe face characteristics, the concept of second-order eigenfaces is 
proposed according to our observation and derivation. At first, the first-order eigenfaces and 
P 10 second-order eigenfaces can be derived from a set of training face images. Then the eigenfaces 
13 can be adjusted with the characteristic of symmetry of faces. With these adjusted eigenfaces, the 

^ adjusted first-order and second-order eigenfeatures can be obtained. The combination of these 

fU eigenfeatures can be used to describe faces. With this description, Euclidean distance can be 

fz used for similarity measurement, 

r 15 

J BRIEF DESCRIPTION OF THE DRAWINGS 

y. Figure 1 is a diagram for generating the light-invariant face descriptor. 

CO Figure 2 is a diagram for generating the view-angle-invariant face 

i" descriptor. 

20 Figure 3 shows the procedure to mirror an eigenface. 

Figure 4 shows the reconstructed faces, the corresponding original faces 
and the second -order residue images. 

Figure 5 shows the procedure for computing the first -order feature ^ ) 



25 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

According to the present invention, first-order Eigenface and second-order 
Eigenface are obtained. 

In Figure 2, the face pictures la, 2a, 3a, 4a, 5a and 6a shown in the middle 
row are original face images captured by a digital camera without any particular 
30 processing. The face picture la can be expressed, by using one dimensional vector, 
as an original face image <f> l . Similarly, face pictures 2a, 3a, 4a, 5a and 6a can be 
expressed respectively, as original face images <& 2 , <J> 3 , 3> 4 , <E> 5 and <I> 6 . 
Generally, the original face image ia is expressed as <&, . 

In the example shown in Figure 2, when one dimensional vectors of the six 
35 original face images are added and the sum is divided by six, an average image T is 
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obtained, as given below. 

¥ + <|> 2 + 0 3 + 0 4 + 0 5 + <D 6 )/6 

A difference between the one dimensional vector of the face picture la 

and the average image *F is expressed as T x m . A similar differences r 2 0) , r 3 0) , 

5 r 4 (1) , r 5 0) , r 5 (1) can be obtained for other face pictures 2a, 3a, 4a, 5a and 6a, 
respectively. Thus, the following equations are obtained. 

r 2 (1) = <e> 2 - *f 
r 3 (1) =<d 3 -t 
r 4 (,) =o 4 -t 
r 5 m =o 5 
r 6 (1) = <P 6 -¥ 

As will be described later, the difference I"/ 0 will be used as a base for 

15 obtaining the first-order feature component vf, (1) . Similarly, r 2 (1) , r 3 (1) , r 4 (1) , 

r s (1) , r 6 (1) will be used as bases for obtaining the first-order feature components 

w 2 G) , w 3 (1) , w 4 m , w 5 0) and w 6 m . In these formulas, the superscript (1) indicates 
that the formulas are related to the first-order feature. 

When the differences r, 0) , r 2 (1) , r 3 (!) , r 4 (1) , r 5 (1 \ r 6 (1) are processed in 

20 a low pass filter, reconstructed matrixes f^, f 2 0) , f 3 0) , f 4 (I) , f 5 0) and f 6 0) are 
obtained. 

In Figure 2, the face pictures lb, 2b, 3b, 4b, 5b and 6b shown in the top row 
4 




are reconstructed face images obtained by adding the. average image *P to each of 



the reconstructed matrixes f, (1 



J-3 > l 4 



and T 6 U 



The 



m 10 

W 



reconstructed face images lb, 2b, 3b, 4b, 5b and 6b can be expressed as O,, 0 2 
3 , 4 > ® s and ^ 6 respectively, as given below. 



£ 4=¥ + r 4 (5) 

Since the reconstructed face images contains data that have been processed 
in the low pass filter so that the high frequency information is removed, the face 
information will not vary even if the angle of the face should be changed slightly. 

In Figure 2, the face pictures lc, 2c, 3c, 4c, 5c and 6c shown in the bottom 
row are residue images Ti t2) , which are obtained by subtracting the reconstructed 
face images from the original face images. Thus, the following equations can be 
obtained. 

T 1 (2) = = r t (1) -f, (1) 

r 2 !2) =$ 2 -<i> 2 = r 2 (1) -f 2 (1) 
r 3 (2) =$3-<i>3 = r3 (1) -f 3 0) 
r 4 (2 > =o 4 -<i) 4 = r 4 (1! -f 4 (1) 
r 5 (2) =o 5 -o 5 -r 5 (1) -f 5 G) 
r 6 12) =$ 6 -6 6 = r 6 (1) -f 6 (1) 

During the process of obtaining the residue images, the brightness 
information, caused by the lighting system, contained in the original face image and 
the same brightness information contained in the reconstructed face image are 
counterbalanced by the subtraction. Thus, the residue images will not be 
influenced by the lighting system. 

The present invention will be further described using general formulas. 



Consider an image 1 among a collection of M images (in the above 
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example, six images)(step 110 in Figure 1), where 1 is a one dimensional vector of 
raster-scanned image, define ¥ as the average image (step 120 in Figure 1): 

1 M 

Every image differs from the average image by a vector r i (,) = <E>, - *F (step 
130 in Figure 1). The covariance matrix of the data is thus defined as: 
Q = A™A« )T (2) 

where A" = )f«% m . .T u m [ 

This is shown in step 140 in Figure 1. 

Note that #has dimension wh xwh where wis the width of the image and 
h is the height. The size of this matrix is enormous, but since we only sum up a 
finite number of image vectors M, the rank of this matrix can not exceed M-l. We 

note that if v/^is the eigenvector of A (l)T A m (i=l,2,...,M) (step 150 in Figure 1), 

then A mT A a \ m = 1,° ty l) where A t m are the eigenvalue of A (lf A m , then 

A mT v^ are the eigenvectors of A (1) A 0)T as we see by multiplying on the left by 
A w in the previous equation: 

But A m \4 (1) isonlyof size MxM . So defining the eigenvector of A m A m we 
have 

The eigenvalue is the variance along the new coordinate space 
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spanned by eigenvectors w, a) (step 160 in Figure 1). From here on we assume that 

the order of i is such that the eigenvalues /L. (1) are decreasing. The eigenvalues are 
decreasing in exponential fashion. Therefore we can project a face image r (1) onto 
only M l «M dimensions by computing W m = \v k m j where w k m =u k wT T m and 

5 \<k<M^. w k 0) is the k-th coordinate of r (I) in the new coordinate system. In 

Z this context, W 0) is called first -order features. The vectors u k l) are actually 

flj images, and are called first-order eigenfaces (in general, it is called eigenfaces in 

m other documents). Let U 0) = ...u M ™] (step 170 in Figure 1), then 

d 

f w u =U «f T M ( 3 ) 

fy 10 This is shown in step 180 in Figure 1. 

Figure 3 shows the procedure for computing first-order features W 0) . In 

the figure, eig(B,i) is the function to compute the i th largest eigenvalue and its 
H corresponding eigenvector of matrix . 320 is re-shaping, 340 is mirroring and 360 

is re-shaping. 

15 From Eq.(3), an interesting byproduct is that we can get a reconstructed 

matrix from W m . Since t/ 0) is an M x xP matrix, we cannot get its inverse. 

However, we can use its pseudo inverse to approximate its inverse. Let be the 
pseudo -inverse of U mT , then 

fv=U m+ W m (4) 

20 where r (,) is the reconstructed matrix from W {X) and U m . Figure 2 gives some 
original face images and their corresponding reconstructed face images with 

M, =25. The top images are the reconstructed face images^ =f 1 (1) + x F), the 

middle images are the corresponding original face images, and the bottom images 

are the residue images (r/ 2) = r/ l) - f, (1) ). From the reconstructed matrices, we 
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can find that the details of the faces are lost. That means what W m describe can be 
looked as low-pass filtered images. The residue images are the corresponding high- 
pass filtered images. Observing the reconstructed matrices, some images cannot be 
reconstructed on a reasonable resolution which indicate that first -order features 
5 cannot describe these images well. The worse the reconstructed matrices is, the 
more information is reserved in residue image. Since these residue images still 
contains rich information for the individual image, face feaUires should be extracted 
from these residue faces again. 

H 10 Let A m = |r i (2) r 2 (2) ...r M (2) ], A™ be the eigenvalues of A {2f A m and v, (2) be the 

Q 
Q 

m corresponding eigenvectors of A (2)T A a) . Then A m * A^v™ = ti?v?-\ Based on the 

a 

yQ discussion above, the eigenvectors of ^ (2) ^ (2)r are u {2) = A™v™ . Therefore we can 

s project a residue face image T (2) onto only M 2 «M dimensions by computing 

K? W m =ty k m }, where 

1 15 w™ = u™ t t™ (5) 

and 1 k <M 2 . Since «/ 2) are the eigenvectors of the residue face image, we call 
u k i2) the second -order eigenfaces and w fc (2) the second -order features. 

Let U m = |w, (2 V 2) ...a Mz (2) ], Eq.(5) can be written as 

w {2) =f/ (2)T r (2) 

20 = u (2)T tr m -u 0) *w m ) 

= u {2f \r m -u m+ u mT r m ) 

= (f/c2^ _ u^fjj^-gnf y m (6) 

Let U 2 = (c/ (2)T - U^U^U^ ) , then 
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Since U 2 is a constant transform matrix and it is just calculated once, it will not 
effect the efficiency of computation. The face images can be described with 

n(d>)= [w/ 1 \w 2 (, V..,w M / 1 \w 1 (2) ,w 2 (2) ,... J w jWj <2) ] r ) where 1 <M X '<M X . The 

computational burden does not increase in computing n(0>) compared with only 
computing features from eigenfaces U. For the convenience of discussion, the 
residue images are called second -order residue image and the original face images 
are called first -order residue images (though it seems it is better to call the original 
images as 0 th -order residue images). 

The first-order eigenfeatures describe the characteristics of original images 
and the second-order eigenfeatures depict the characteristics of high-passed images. 
These characteristics can be used for different applications. 

In lighting-invariant case, the view-angles of the images are almost the 
same, but the lighting conditions are different. The light can come from left, right, 
or both. Also the faces can be light or dark. In this situation, the features of images 
should be extracted after filtering. The original face image contains data of 
lighting feature, and the first-order eigenfeature also contains data of lighting 
feature. That means the first-order eigenfeatures are not suitable in this case. 
The second-order eigenfeature is obtained by subtracting the first-order 
eigenfeature from the original face image, thus removing the data of lighting 
feature. Thus, the second-order eigenfeatures have no influence by the lighting, 
and are suitable for use in face recognition that does not influenced by the lighting. 
On the other hand, when we consider view-angle-invariant case, the images of the 
same person are taken under the almost the same lighting condition but the view- 
angles or poses are not the same. In this situation, both the first-order and second- 
order eigenfeatures are important to describe the faces. 

The first-order eigenfeature is obtained by processing the original face 
image through a low pass filter to extract, not the detail, but general feature of the 
face. Thus the first-order eigenfeature can provide approximately the same 
feature even if face images are obtained in different view-angles. Thus, in the 
cases where the view-angles may change, the first-order eigenfeature is used in 
addition to the second-order eigenfeature so as to obtain features which will not be 
influenced by the view-angle. 

Therefore we will use second-order eigenfeatures as face descriptor to 
describe lighting-invariant situation and both of the first-order and second-order 
eigenfeatures as face descriptor to describe view- angle-invariant situation. 



When we consider face description, an important characteristic is 
symmetry of faces. The faces of most of the people are symmetric. Thus we can 
consider to use the mirrored face image to represent the same person. Thus the 
eigenfeatures of the original image and the mirrored image can both represent the 
same person. Therefore, we can use the linear combination of the eigenfeatures of 
the two images to represent the original image. In order to discriminate between 
the original image and the mirrored image, different weights can be used to 
calculate features. 

Suppose <& be the original image and O' be the mirrored image of <I> . 
Then the adjusted eigenfeature should be 

and 

^ 2) (0) = W , C2) (0) + C - W; (2) (0') 

where 0 < c < 1 is the weight for adjustment. In order to simplify the computation, 
the computational order can be rearranged. 

Since w 1 m (V)=u t mT V m = u l mT r m , 

= («/ I)r +c . M , (1),r )r (! > 

where w, 0) is the mirrored eigenface of . Figure 1 shows the procedure to 
get from . With the same method, we can obtain 

w ll) =u mT r™ +c-u« )T f& v (r™) 
= f/ (1)r r (I) +c-fiip(i7 (If )r <!) 
= (t/ (I)r +cfii P (i7 (1)r ))r (1) 

= U:T W (8) 

and 
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w m = u 2 T Y m + c.u 2 r fii P (r (,) ) 
= u/T m +c-m P (u T 2 )r m 
= (u 2 T +c-m v (u T 2 ))r m 
= u 2 r m (9) 

where mirrorO is the mirror function to mirror matrix in left/right direction. With 
Eq.(8) and Eq.<9), the first-order eigenfaces and second-order eigenfaces and can 
be adjusted to U\ and Uz ■ 

The eigenfeatures discussed above can be used to describe huma faces 
effectively. However, if the eigenfeatures are looked as features for face recognition 
or related application, the eigenfeatures should be normalized since the range of the 
eigenfeatures are not the same. 

One of the reasonable normalization method is to divide the eigenfeatures 
by the corresponding standard deviation of eigenfeatures in the training set. Let 

crf^ be the standard deviation of w ; 0) in the training set, where j=l,2 denotes the 

first- or second-order eigenfeatures. Then normalized eigenfeatures of image O are 

The similarity of face images is simply defined as the weighted distance 
between normalized projections: 

^(*,*j=ElK1 ff . < ' > (*.)-*. < ' > (*')| + 2KK*/ 2) («i',)-<i ; ; 2> (*,)l co) 

1=1 j=i 

If a { p = 0, the similarity of face images will be measured only with second 

-order eigenfeatures for lighting-invariant face description. For view- angle- 
invariant case, both first- and second-order eigenfeatures are necessary. The 
weights are chosen as follows: 




The weights can also select other numbers according the applications. In 
fact, the weights can be embedded into adjusted eigenfaces to reduce the 
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computation, and the normalization parameters can also been moved to eigenface 
matrix. In order to save the storage, the eigenface matrix and eigenfeatures should 
be quantized into integers from floating points. 

Let w, 0) = Round a™ ^R U) > where R U) = 7 , and RoundQ is 



max 



the round function, and /=1,2. The eigenfeatures can be quantized with 

f- 1 28, if RoundQ 0) «, (/) r (I) ) < - 1 28 

w f 0) (0)=J 127, // Round(rf O) ^^r 0) )>127 , (10-1) 

[ Round(</ 0) *?, o) r a) )i otherwise 

where d U) = j ,r t is the images in the training set. The 

feature set can be written as W 0) and W m = {wf ) }. Therefore, each 

eigenfeature can be quantized to 8 bits. 

Further, in order to reduce the size of total eigenfeatures, the eigenfeatures 
can be re-quantized according to their standard deviation of the training set. This 
can be considered as the trade-off between the size of face descriptor and the 
retrieval accuracy. According to our experiments, the bits for each eigenfeature can 
be allocated as follows: 

Nj n - roundf log 2 [ — °) , ..\ x 16 

where min(a 0) ) is the minimum of the standard deviations of the j-th order 
eigenfeatures. Though different eigenfeature allocate different bits, the range is 
still remains [-128,127]. With this quantization strategy, the size of face descriptor 
can reduce about 40% while the retrieval accuracy drop about 1% to 2%. If the 
original quantization strategy is acceptable, then the further quantization is not 
necessary. Here a normalizing and clipping procedure example is given to further 
quantize and compact the feature from the equation (10-1) and each eigenfeature 
can be represented as unified 5 bits: 
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[ 0, if w i /Z<-\6 

w 1 =| 31, // w t IZ>\5 

\round(w t /Z + 16), otherwise 

where Z= 16384*8 is the normalization constant from our experience. 

With the above formulae, the distance between two face images $> x ,® 2 for view- 
5 angle-invariant description, can be calculated as 

^(^: , <o 2 ) = s h a) )- (<i> 2 )|| + 1;||-/ 2) ) - -; 2) (^>4I od 

□ and the distance distance between two face images < 5 1 ,3> 2 for lighting-invariant 



a 



description, can be calculated as 



yj 10 In order to further compact the face descriptor, variable length coding can be used to 

~ the quantized eigenfeatures. The Huffman coding and arithmetic coding can be used to do the 

rsj work. 

y= The code tables can be computed as follows : 

1 ) for the images in the training set, calculating the eigenfeatures; 
y* 15 2) quantizing the eigenfeatures with the stretegy discussed above; 

3) for the quantized eigenfeatures with the same bit allocation, computing the probability of 
each quantization level; 

4) calculating the variable length codes for all quantization levels; 

5) calculating all code tables with the steps 3) and 4) for all bit allocation cases. 

20 With this method, the face descriptor can be further compacted. Correspondingly, The 

coded face descriptor should be decoded before the distance between two face descriptors can 
be calculated. 

According to our observation, the first-order eigenfeatures itself can also be used to 
describe view-angle- and lighting-invariant face features. The view-angle-invariant face features 

25 can be the eigenfeatures extracted from the eigenfaces corresponding to the top N largest 
eigenvalues. The lighting-invariant face features can be the eigenfeatures extracted from the 
eigenfaces corresponding to the top k-th to N-th largest eigenvalues where 0<k<N. In general, 
4<k<10 and 4(KN<60. We recommend the use of k = 8, N = 40 and total eigenface number is 
48. Figure 4 and Figure 5 show, respectively, a flow chart for extracting eigenfeatures that 

30 will not be influenced by the lighting, and a flow chart for extracting eigenfeature that will not 
be influenced by the view-angle. 
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In Figure 4 showing the method of extracting eigenfeatures that will not be influenced 
by the lighting, the following steps are provided. 
Step 410: Calculate the adjusted second -order eigenface matrix. 
Step 420: Get the adjusted second -order engenfeatures. 
5 Step 430: Select features to describe faces from the adjusted second -order eigenfeatures. 
Step 440: Code the quantized eigenfeatures with VLC. This step can be omitted. 

In Figure 5 showing the method of extracting eigenfeature that will not be influenced 
by the view-angle, the following steps are provided. 
Step 5 10: Calculate the adjusted first -order eigenface matrix. 
U 10 Step 520: Calculate the adjusted second -order eigenface matrix. 
h J Step 530: Get the adjusted first -order eigenfeatures . 

7= Step 540: Get the adjusted second -order eigenfeatures. 

O Step 550: Quantize the adjusted first-order eigenfeatures and the adjusted second -order 

In eigenfeatures. 

U) 15 Step 560: Select features to describe faces from the adjusted first -order and second -order 
f 5 eigenfeatures. 

pj Step 570: Code the quantized eigenfeatures with VLC. This step can be omitted. 

This invention is very effective for describing human faces. Since the second-order 
;S eigenfaces can be calculated only once with the training face images, the second-order features 

I s * 20 can be obtained as efficient as first-order features. Since detailed information can be revealed 
with high-order eigenfeatures, the performance of the combination of first-order and second- 
order eigenfeatures is better in view-angle-invariant face description compared with the same 
number of first -order eigenfeatures. The second-order eigenfeatures has much better lighting- 
invariant face description capability compared with the first-order eigenfeatures. This invention 
25 is very effective and efficient in describing human faces. The description method can be used in 
internet multimedia database retrieval, video editing, digital library, surveillance and tracking, 
and other applications using face recognition and verification broadly. 

The present disclosure relates to subject matter contained in priority Japanese Patent Application 
No. 2000-385126, filed on December 19, 2000, the contents of which is herein expressly incorporated by 
reference in its entirety. 
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